Web-scale Topic Models in Spark: An Asynchronous Parameter Server

In this paper, we train a Latent Dirichlet Allocation (LDA) topic model on the ClueWeb12 data set, a 27-terabyte Web crawl. We extend Spark, a popular tool for performing large-scale data analysis, with an asynchronous parameter server. Such a parameter server provides a distributed and concurrently accessed parameter space for the model. A Metropolis-Hastings based collapsed Gibbs sampler is implemented using this parameter server achieving an amortized O(1) sampling complexity. We compare our implementation to the default Spark implementations and show that it is several orders of magnitude more scalable without sacrificing model quality. A topic model with 1,000 topics is trained on the full ClueWeb12 data set, uncovering some of the prevalent themes that appear on the Web.